Compiling the uncompilable: A case for shell script compilation

Shells, as command interpreters, are the classical way for humans to interact with computing systems, and modern shell features have extended this basic functionality with higher-level programming language constructs. Although implementing compilation in these shell languages is generally unprofitable and intractable, many advantages, such as isolation, filesystem abstraction, security, portability, parallelization and locality optimization are possible, using standard compilation techniques. While compilation is not possible for all scripts, there exist shell scripts of a class that are, in practice, both profitable and tractable to compile and execute. This class of scripts is prevalent in the scientific computing community, where scripts are commonly used to automate data processing sequences. We describe a prototype shell compilation and implementation for these scripts, noting advantages and challenges, and illustrating the significant performance potential available. Our results show that shell compilation is a viable means of automatically identifying and exploiting high-level program parallelism using existing sequential script specification and without requiring reimplementation.